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Permutation  Tests  for  k-Sample  Binomial  Data 
Andrew  P.  Soms* 

Summary 

Exact  k-sample  permutation  tests  for  binary  data 
for  three  commonly  encountered  hypotheses  are  presented. 

The  asymptotic  distributions  of  the  test  statistics  are 
derived  and  some  numerical  examples  presented.  Sample 
size  guidelines  are  given  for  computer  implementation  of 
the  tests. 

1.  Introduction 

Let  Y.  . , 1 < i < r,  1 < ,j  < k. , be  independent 

Bernoulli  random  variables  with  parameters  and  observed 

_ k. 

r i v 

values  y . . and  let  m,  = E E y . . , m^  = E k . - m,  and 

1J  1 i=l  j=l  ° l 1 1 

r 

k = E k^.  The  three  hypotheses  of  interest  are  HI:  p.  = p, , 

1 

• • p-1 

2 < i < r,  against  the  2 -1  alternatives  that  p^  > p^  if 

icA  and  p^  = p-^  if  icAc,  where  A is  any  subset  of  {2,...r}, 

H2 : p.  = p.  for  all  i / j against  the  alternatives  that  a 
J 

specified  subset  of  pairs  (i,o)  are  different,  and  HJ: 

?1  = ?2  = *’*  = Pr  a6ainst  the  alternative  that 

P1  — p2  — ’**  — Pr  at  least  one  strict  inequality. 
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In  the  usual  terminology,  HI  corresponds  to  the  one-sided 
comparison  of  r-1  treatments  to  a control  with  upward 
shifts  in  the  p^'s  being  of  interest,  H2  to  the  one-way 
ANOVA  for  0-1  data,  and  H3  to  testing  for  the  presence  of 
a dose  response  (here  it  is  assumed  that  the  labelling 
index  i corresponds  to  the  i—  dose  level). 

First,  consider  HI,  the  problem  of  comparing  r-1 
treatments  to  a control  with  a one-sided  alternative 
being  of  interest,  similar  comments  applying  to  H2.  Some 
examples  are:  advertising  effectiveness  of  r-1  new  methods 

against  the  existing  or  standard  and  drug  side  reactions 
of  r-1  new  drugs  against  the  standard.  For  r > 2 there 
has  been  no  exact  treatment  of  this  problem — some  possibilities 
that  have  been  given  in  the  literature  are  the  likelihood 
ratio,  one-way  normal  ANOVA,  or  the  Kruskal-Wallis  test. 

In  addition  to  being  approximate  (because  of  ties),  these 
tests  have  another  drawback — they  merely  test  the  null 
hypothesis  of  equality  but  give  no  further  information  in 
case  of  rejection,  i.e.,  they  do  not  tell  which  alternative 
to  accept.  If  Scheffe  type  procedures  are  used  for  the 
ANOVA  or  Kruskal-Wallis,  then  even  in  the  continuous  case 
they  are  conservative — e.g.,  comparison  of  the  S-intervals 
to  the  multivariate  t intervals  for  the  normal  ANOVA  shows 
the  S-intervals  to  be  longer.  Here,  of  course,  the 
additional  problem  of  discrete  data  exists.  Thus  F-type 
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tests  are  not  appropriate  for  this  problem  nor  for  H2. 

If  unbiasedness  for  HI  and  H2  is  defined  in  a reasonable 
manner,  it  follows  from  the  results  of  Lehmann  [9]  that 
no  unbiased  test  exists,  and  hence  consistency  is  the 
best  one  can  hope  for. 

H3  is  the  hypothesis  of  interest  in  quantal  assay. 

It  is  also  of  interest  in  the  toxicological  testing  of 
drugs  where  often  the  criterion  of  toxicity  is  whether  a 
dose  response  exists.  A common  situation  in  drug  testing 
is  to  have  a control  group  and  three  dose  groups — low, 
medium,  and  high — of  randomly  selected  small  test  animals. 
After  a period  of  administering  the  drug, the  number  of 
deaths,  unfavorable  reactions,  or  some  other  appropriate 
toxicity  criterion  is  observed  for  its  presence  or  absence 
in  each  animal  and  it  is  desired  to  test  for  dose  response. 
In  the  continuous  case  such  tests  are  known  (see,  e.g., 

[8],  pp.  232-8),  but  here  again  the  problem  of  ties  exists. 
The  simple  permutation  test  proposed  here  is  shown  to  be 
consistent. 

The  purpose  of  this  paper  is  to  present  a unified 
approach  to  HI,  H2,  and  H3  based  on  permutation  tests. 


2.  Permutation  Tests  for  HI  and  H2 


In  all  that  follows  the  notation  in  1.  will  be  used. 

r ku 

Under  HI,  H2,  or  H3,  Y = S 27  Y.  . is  a complete  and 

i=l  j=l  10 
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sufficient  statistic  (see,  e.g. , [3],  p.  31  and  p.  19) 
and  hence  any  a-level  test  also  has  conditional  level  a 

and  so  it  is  sufficient  to  specify  a conditional  test. 

k. 

1 

The  conditional  distribution  of  Z.  = E Y.  .,  1 < i < r, 

1 J=1  - 


given  Y = is 


r 

n 

1 


a 


P[Z.  = n. , E n.  = m,  , 1 < i < r]  = , 

1 iii  1 - - ®0+mll 


m. 


and  the  carrier  set  is  given  recursively  by 

max(0,kr-mo)  < np  < min(m]L,kr) 

• 

t t . r 

max(0,Ek--  E n . -m  ) < n.  < min(m,-  E n. ,k.) 

d 1 d+1  “ J 1 d+1  1 J 


(2.1) 


(2.2) 


r r 


max(  0,Ek.  -En.  -m  ) < n,  < min(m,  -En.  ,k,  )‘. 

2 ^ u x 2 x 1 

These  inequalities  for  n.  are  obtained  by  requiring  that 
the  interval  for  n^+^  be  nonempty,  which  is  the  case  if 
and  only  if  each  of  the  two  expressions  on  the  right  is 
greater  than  or  equal  to  the  two  expressions  on  the  left. 
Elimination  of  redundancies  yields  the  interval  for  n.. 

V 

The  interval  for  n is  deduced  directly  from  the  con- 
r r r 

straints  E n.  = m,  and  E(k.-n.)  = m . Note  that  if 
^ 1 1 ^ 1 1 0 

yi  = (yii*  * * * ,yik.  ^ ’ y = ) and  if  each  permutation 
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of  y has  probability  l/(m  +1^)1  = 1/kl , then  the  conditional 
distribution  is  (2.1).  This  important  fact  will  allow  the 
use  of  the  Wald-Wolf owitz-Noether  Theorem  to  obtain  the 
limiting  distribution  of  the  test  statistics.  For  the 
sake  of  completeness  the  statement  is  given  here  (see  [3], 
p.  237  for  the  details).  Let  Cpn*****0^  and  ain»***»ann 
be  two  sequences,  each  containing  at  least  two  distinct 
values,  such  that  for  all  r = 3,  4,  . . . the  condition  W 

i 11 

-T-5 7575  ' °0)  (2.5) 

L-  £ (c.  -c  fy 
n m n'  J 

and  for  some  r > 2 the  condition  N 


(2.3) 


• £-,  a . -a 
i=l  in  n 

n 0 ] 

] 


(2.4) 


is  satisfied,  where  c = 2 c /n  and  a = 2 a /n  . 

n i=l  in  n i=l  in 

Let  Xn  = (Xj,  - - . , X^)  be  a random  variable 

which  takes  on  each  permutation  of  (a,  ,...,a  ) with 

n m nn 

probability  1/n!  and  let  Ln  = £ cinXi  * Then 

n n 

E{Ln)  - ( £ c.  £ a.  )/n  and 
i=l  in  j=l  Jn 


Var{L  } = — i-r-  £ (c.  -c  )2  £ (a . -a  )^  = o‘ 

nJ  n-1  • , v in  n'  . v in  n'  1 
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(Ln-ELn)/oL  converges  in  distribution  to  the  standardized 
n 

normal . 


Let  Z = max 


Vsi 


2<i<r  UAj+l/kj) 


Yj2'  where  = Z. 


1 < i < r.  The  descriptive  level  associated  with  the 


. th 

1 — group 


yi-yl 


ki 


is  PCZ  > TT^'  where  y-:  = S y.  ,/k-  , 

(IA^IA-l)172  1 0 = 1 1J  1 

i < i < r (PC*]  stands  for  (2.1)).  For  any  choice  of  the 

p -1 

r-1  possible  descriptive  levels,  Hi  or  one  of  the  2 - 1 

alternatives  is  accepted  as  follows:  if  the  descriptive 


level  associated  with  the  j—  group  is  less  than  or  equal 
to  the  chosen  descriptive  level,  the  j—  group  belongs  to 
the  set  different  from  the  control,  otherwise  it  is  in 
the  set  equal  to  the  control.  This  type  of  simultaneous 
permutation  test  has  been  suggested  in  Miller  ([10],  p.  180) 
for  continuous  data  and  equal  sample  sizes.  Generally, 
except  for  small  m^  or  m , the  test  has  to  be  carried  out 
on  a computer  and  (2.2)  is  very  convenient  for  a nested 
do-loop  procedure,  each  point  in  the  carrier  set  being 
tested  r-1  times  and  the  probabilities  summed  in  the  inner- 
most do-loop.  The  number  of  cases  in  the  carrier  set  for 
four  groups  of  equal  sample  size  as  a function  of  the 
group  size  is  given  in  the  table. 
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L.  Number  of  Gases  for  Four  Groups  of  Equal  Size  with  Half  Ones 


Group  size 

5 

'10 

20 

30 

40 

80  ' 
100 


Number  of  cases 
146 
891 
6 181 
19  871 
45  961 
354-  321 
686  901 


Consider  now  the  asymptotic  behavior  of  the  permutation 

test  under  the  assumption  that  lim  kn/k.  =c.  >0,  2<i<r. 

k±— » ill  - - 

Let  L.  = — £ , 2 < i < r, 

(iAi+iA1)1/2(  s & (yi1-y)2/(k-i))1/2 
1 1 i=l  j=l 

k. 

_ r i 

y = E E y.  . A*  It  follows  that  E{L.}  = 0,  Var  {L.  } = 1, 
i=l  o=l  J 1 x 

Corr(Li,Lij)  = (k.kJ/((k1+k.)(k1+kij)))1/2,  i / j (C31,  p.  240). 

By  a slight  extension  of  the  argument  given  on  pp.  244-5  of 

[3]  it  follows  that  under  HI,  H2,  H3  or  the  three  alternatives 

(more  generally,  simply  under  the  assumption  that  there  are 

r groups  with  p^  the  parameter  for  the  i—  group)  (2.4)  is 

satisfied  with  probability  one  for  r = 5.  Also,  it  is  seen 
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by  direct  verification  that  (2.3)  holds  for  the  coefficients 
of  each  permutation  statistic  F-Z^,  2 < i < r,  so  each 
is  marginally  normal.  By  considering  arbitrary  linear 
combinations  of  the  scaled  ([3],  p.  241)  Z.  -Zj  , 

2 < i < r,  it  follows  by  the  Wald-Wolfowit z-Noether  Theorem 
that  the  limiting  joint  distribution  of  . . . ,Lr  is 
asymptotically  normal  with  mean  0 and  variance-covariance 
matrix  R = [p^],  p±.  = l/(l+ci)  (1+c..  )1/2  if  i / j , 

= 1 if  i = j (it  is  checked  directly  that  (2.3)  is 
satisfied).  Since  the  1^,  2 < i < r,  differ  only  by  a 

i/2 

scaling  factor  from  the  (Zj-Z^/fl/k.+l/kj)  , the  permutation  test 
may  be  equivalently  performed  with  the  Lj's.  So  an  approximation 
to  the  descriptive  level  associated  with  the  i—  group  is 


P 


Max  X. 
2<j<r  ^ 


yi-y1-i/(2Ei) 

(lA^iA^17^  2 (yiry)2/(k-i))1/2 

1 1 i=i  j=i 


i-p 


X.  < 
J - 


yi-yi-1/(23ti) 

— : — f 


2 < j < r 


(lAj+lAjO 


1/2 


( 2 Z (yii-y)2/(k-l))1/2 
i=l  j=i  10 


where  F = (k^+k-^)/2,  the  X^  are  multivariate  normal  with 
means  0 and  variance-covariance  matrix  R (here  it  is  assumed 


that  c^  = k1Ai) » and  a continuity  correction  has  been  used. 

This  probability  may  be  obtained  by  linear  interpolation  in 
the  table  of  Gupta  [4]  for  the  case  of  equal  kVs  (for  the  case 
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1 


of  unequal  k^'s  a procedure  is  given  below).  As  an  example, 

for  four  groups  of  forty  each,  with  half  ones,  such  that 

max  (Z.-Z,)  = 10,  the  exact  probability  associated  with 
2<o<4  J ± 

this  value  is  .045,  while  the  approximation  from  Gupta's 
tables  (using  2.12  as  the  value)  gives  .044. 

The  above  results  will  now  be  used  to  give  a brief 
discussion  of  consistency.  Assume  that  randomized  tests 


are  used  to  achieve  a significance  level  a and  that  ta(Z) 
is  the  randomized  cut-off  point  for  the  L^'s.  By  the  above, 
ta(Z)  converges  almost  surely  to  d , where 
P[Xi  < d , 2 < i < r]  = 1 - a.  Hence  the  limiting  form 


of  the  permutation  test  is  to  say  p^  = p^  if 


yi“yl 


r i 

(l/k.+l/k  )1/2(  E E (y  -y)2/(k-l))1/2 

1 i=l  0=1  1J 


< d^  and  if  not, 


to  conclude  p^  > p^. 


V4 


(lAi  + l/k1)1/2(  S E (Y.,-Y)2/(k-l)): 

1 1 i=l  o=l  J 


r ki 


2<i<r,  Y = Z Y /k  Y = Z Z Y A,  Y = (Y  Y Y ) 

J = 1 3 i=l  j=l  3 11  u r»  r 


i.  e. , the  T^’s  are  the  unconditional  permutation  statistics.  By 
considering  linear  combinations,  it  is  seen  that  the  limiting 
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7i_7l_<;pi’p  l) 

joint  distribution  of r 

x*  i 

d.(lAi+lA1)1/2(  2 S (Y.  ,-Y)2/(k-l))1/2 
1 1 1 i=l  j = l 


2 < i < r,  is  the  multivariate  normal  with  0 means  and 
unit  variances  and  correlations  that  are  functions  of 
c^,  2 < i < r,  and  p^,  1 < i < r,  where  the 

di  = ”c)  are  positive  constants 

with  c the  almost  sure  limit  of  ( E ^ (Y.  .-Y)2/(k-l)  )1/22 

i=l  j=l 


and  q^  = 1-p^,  1 < i < r.  Then,  by  the  same  argument  as  in 
Hoeffding  ([6]  p.  171)5  the  limiting  probability  of  any  one 
of  the  2 decisions  is  (here  (X  , ...  ,X)  has  the  limiting  distribution) 

d p . -p, 

lim  P[X,  £ ™ 2 < i < r], 

ki-*®,l<i<r  1 < 1 dj(lAi+lA1)i/^ 


where  the  d^  = ((c^p^q^+p1q1)/(c^+l))'1'//^  are  positive 
constants.  So  if  P2  > p^t...)  pp  > p^,  then  with  limiting 

probability  1 they  will  be  identified  as  being  bigger  than 
P-p  The  limiting  probability  of  correct  decision  is 

P[Xr1+l  < da/dr1+l’*** ,Xr  - do/dr]* 


The  procedure  and  results  for  the  two-sided  case  and 
the  one-way  AI10VA  are  very  similar.  For  the  two-sided  case 
of  treatment  to  control  comparisons  absolute  values  are 
used  in  the  definition  of  Z and  for  the  constants  for  which 
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the  descriptive  levels  are  to  be  found.  The  approximations 

to  the  descriptive  values,  for  the  case  of  equal  treatment 

group  sizes,  can  be  found,  just  as  before,  by  linear 

interpolation  in  the  tables  of  Dunn,  Kronmal , and  Yee 

Cl].  For  unequal  treatment  group  sizes  the  interpolating 

scheme  given  in  Dutt,  Mattes,  Soms,  and  Tao  [2]  can  be 

used  for  both  the  tables  in  [1]  and  [4],  Briefly,  the 

procedure  consists  of  calculating  an  effective  treatment 
r 


group  size  k = £ k./(r-l)  and  using  It  as  the  equal  treat- 

i=2  1 

ment  group  size  to  obtain  an  approximation  as  discussed 
above.  The  comparisons  in  [2]  of  the  approximation  to  the 
exact  values  indicate  that  the  approximation  is  adequate  for  all 


practical  purposes. 

For  all  .comparisons,  consider 
Z.-Z.  I 

Z = max  = — “ — * —775,  i / j,  and  compare  this  to  the 

i,j  (lA^iA^r72 

corresponding  observed  quantities.  The  approximation  to 
the  exact  descriptive  level  (after  division  by  the  sum  of 
squares  as  before)  can  be  found  in  the  tables  of  Harter 
[5]  for  the  case  of  equal  group  sizes.  In  the  case  of 
unequal  group  sizes,  dropping  the  continuity  correction, 
interpolation  can  be  used  on  the  degrees  of  freedom,  as 
discussed  by  Keselman,  Toothaker,  and  Shooter  [7],  where 
one  of  the  methods  mentioned  is  to  use  the  harmonic  mean 
of  the  sample  sizes  to  obtain  an  effective  equal  sample 
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size.  Note  that  the  tabulated  values  in  [5]  must  be  divided 

by  /5.  As  an  example,  for  four  groups  each  of  size  40  with 

15>  18,  20,  and  27  ones,  the  exact  probability  is 

P[Maxlz.-Z.|  >_  12]  = .049,  while  the  approximation  from 
i<j  J 

[53  (using  2.56  as  the  value)  is  .051. 

Numerical  examples  of  HI  and  H2  are  now  given.  Suppose 
that  three  new  advertising  techniques  are  to  be  compared  to 
the  existing,  each  method  given  to  20  of  80  randomly  chosen 
people  and  the  response  is  "like  - 1"  or  "dislike  - 0". 
Control  gets  6 ones,  the  first  method  14,  the  second  13, 
and  the  third  7-  Carrying  out  the  exact  test,  it  is  found 
that  for  a = .051,  the  first  and  second  methods  are  better 
than  the  control.  For  the  one-way  ANOVA,  suppose  there  are 
four  different  therapies  for  a disease,  with  each  either 
having  or  not  having  a side  effect  on  a subject.  It  is 
desired  to  test  whether  all  are  the  same  with  respect  to 
incidence  of  side  reactions.  The  experimental  set-up 
is  as  above,  with  1 standing  for  the  absence  and  0 for  the 
presence  of  a side  reaction.  Therapy  1 gets  6 ones,  2 gets 
8 ones,  3 gets  15  ones,  and  4 gets  11  ones.  Then  at  the 
.035  level  it  is  concluded  that  3 and  1 are  different. 
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3 = A Permutation  Test  for  Dose  Response 

A reasonable  parameter  to  measure  the  extent  ox  dose 

r i-1  r 

response  is  A = E (p.-p.)  = E E (p.-p.)  = E (2i-r-l)p. , 
i>j  J i=2  0=1  1 J i=l  1 

which  is  oust  a linear  combination  of  the  pVs,  the  largest 

indexes  receiving  the  largest  weights.  Therefore  the 

permutation  test  here  proposed  is  to  reject  H3  for  large 

r k 

values  of  L = E(2i-r-l)Z. » given  Z = E E Y.  ..  As 
1 11  i-1  0=1  k 

r 2 r i 

before,  E{L}  = 0 and  Var{L}  = of  - -X-(z:  *2i,'r"^  ) s £(y,-y)2  = 

L k-1  1=1  kl  1=1  1=1  0 

r — r m (1  -— ) Z . Hence  by  the  Wald  - Wolfowitz- 

i x i=1  JCj 

Wolfowitz-Noether  Theorem,  L/cr^  converges  in  distribution  to 
the  standardized  normal  and  this  can  be  used  as  the  asymptotic 
version  of  the  test  (It  is  verified  directly  that  (2.3)  is 
satisfied).  For  four  groups  with  equal  sample  size 

2 20  m^(l-m^/4s) 

s>  = s(4s-l) » which  reduces  to  20/(4 s-1)  when 

m^  = 2s.  As  an  example,  suppose  in  a toxicity  test  40 
animals  are  put  on  each  of  control,  low,  medium,  and  high 
doses  and  at  the  end  of  the  test  there  are  16,  20,  20,  and 
24  deaths,  respectively.  Then  the  exact  probability  is 
P[L/o^  > 1.69]  = .052,  while  the  limiting  normal  value 
with  continuity  correction  is  .049*  corresponding  to  1.66. 

An  approximation  that  works  even  better  is  to  use  the 
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continuity  correction  and  regard  (L-l/23t)/a  as  a t with 

It  degrees  of  freedom,  where  It  = E k./4-»  For  the  above  this 

1 1 

gives  .053  as  the  approximation.  Consider  now  the  question 

of  consistency.  From  above,  L/o^  converges  in  distribution 

to  the  standard  normal  and  it  is  known  that  the  same  is 

true  for  the  unconditional  random  variable 

r ki 

E(2i-r-l)  E Y.  - A ,-A 

j=l  .1=1  1J  1 


( £(2i-r-l)2  S^)1'2 
i=l  Ki 


and  hence  the  limiting  distribution  of 


E(2i-r-l)  E1  Y.  -A.-A 

i=l  ,i=l  1J  1 

— 

° L(  s 2 Yii> 

i-l  3*1  1;! 


is  normal  with  0 mean  and  positive 


r ki 


P 2 P 

variance  oc,  where  cr-f(  E E Y.  .)  is  of  with  m,  replaced  by 

i=l  0=1  10  1 

r ki 

E S Y. ..  Therefore  the  limiting  probability  of  rejection 

i-l  0=1  1J 

r ki 

E (2i-r-l)  E Y.  -A.-A 
i=l  .i=l  10  1 “a  A 

^72 ~ 


is  lim  P 
k 


57  -c.  , 2<i<r 


c(lA) 


c (1A) 


172 


where  z is  the  standardized  normal  upper  100a — percentile  and  c 
a 

and  c‘  are  positive  constants,  and  therefore  the  test  is  consistent  for 
positive  A . 

For  r = 2 the  test  given  here  reduces  to  Fisher's  exact  test. 
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4.  Concluding  Remarks 

The  tests  here  described  are  suitable  for  computer 
implementation.  For  the  case  of  equal  sample  sizes  and 
an  equal  split  of  o's  and  l's  the  approximate  limitations 
on  each  group  size  are  100  for  r = 4,  40  for  r = 5,  20 
for  r = 6 and  10  for  r = 7»  for  the  case  of  less  or  more 
ones  these  limits  increase,  because  it  is  to  be  expected 
that  the  carrier  set  is  largest  for  an  equal  split  of  o's 
and  l's.  Fortunately,  for  the  case  of  toxicological 
testing  often  r = 4,  and  hence  it  should  almost  always  be 
possible  to  carry  out  the  exact  test.  For  r > 8,  hopefully 
power  considerations  would  lead  to  sample  sizes  for  which 
the  asymptotic  versions  of  the  permutation  tests  could  be 
used  whenever  m-^  or  mQ  are  moderate,  the  exact  tests  being 

run  only  for  small  m,  or  m . 

1 o 

Listings  of  short  computer  programs  that  carry  out  the 
tests  for  HI,  H2,  and  H3  are  available  on  request  from  the 
author.  These  programs  were  used  for  the  examples  dis- 
cussed above. 
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